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Preface 



Oeoaslorimlly novioB evaluators of eduCEtlonal produots do not recognize 
the full scope of their jobi and may be heard to say . .well, thara isn't 
much to do because we won't have a product prototype ready for some time," 
Tha brief dialogue that followB attempte to illustrate that there are a few 
things an evaluator can do before he sees a produot—perhape even too much* 
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PRE-PRODUCT EVALUATIONt 
WHAT TO DO WHILE WAITB^G FOR THE PRODUCT 



Evei Hi, Al, how are you doing on your new evaluation job? Do you like 
it? 

All Yeh; lt*s really fun. The developers I work with are really nice 
people and my job is veiy easy so far. They won't even have a 
produat to test for the ne^rt six months so right now Vm Bpending 
most of my time helping with development. Occasionally I get a 
little bored, but all In all it's pretty interesting and I'll start sonae 
avaluat Ion work In a few months. 

Eves That's strange because my first sIk months here were among my 
busiest, 

Al: Why^ what did you find to do? 

Evei Well, there was a lot of preparation work and then plaimlng to do, I 
was also providing a lot of Information to the developers and also, 
of course, actually doing some evaluation stuaies. 

Al: But how could you do any evaluations without a product? 

Eves Oh, there are several things that may need to be evaluated before 
you start looking at the actual product*' For example, you often 
need to look at the evidence that a strong need exists for the product 
that helps to jtistlfy its development* And sometimes you evaluate 
the adequacy of the goals being developed for the product. You 
might also evaluate the congruence between the goals and needs and 
the product as it evolves. 

All I have been collecting some information for the developers to help 
them write a needs statement and did comment on some of their 
goal statements that seemed a bit idealistic and vague. But that 
didn't take much time. 

Eves in my ease I found it important not only to help the developers col- . 
lect Information about their goals and the need for the product, but 
also to actually conduct an evaluation study of both the need claims 
being made for the system and the worthwhlleness of the goals. In 
one case I used a lot of outside consultant help In reviewing the goals 
Md objectives stataments, but both times sub 
to the developers and had some meetings to discuss the results, 
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AU Didn't the developers feel you were just going over old ground? After 
allj the funding agency bought the needs and goals when they doled out 
the naoney* 

Eve* Somewhat, but I explained that their work was probably p)lng to be 
subjected to this kind of re'^ew later and that I felt It was naLy 
responsibility to give them a foretaste of that and help them revise 
their work now when it was probably easiest, I want them to turn 
out the best possible product, and they may not be able to do that if 
they don^t develop a product that has defensible goals and meets 
some Important educatiorial needs. To a large ejctent, the funding 
agency had bought Intent^;, but now I want to help the development 
staff teke a look at how costly and successful the development effort 
Is likely to be* 

Ah Wellj I suppose you could point out that the product would be more 
expensive if It required costly media or skilled trainers, but it 
sounds as though you are almost being a manager* 

Evsi Not really^ just providing feedback and helping the developers chart 
the progress of their work. Although some evaluators I Imow have 
helped evaluate the actual management of the product development 
effort* 

All Why would they bother to do that? 

Evai Well, administrators sometimes want additional Information on how 
wall the work is. being managed. An evaluation of management 
activities might give some useful Information on whether the work 
Is meeting deadlines, what activities are slowing down pro^essj and 
where the work Is really staying on target. For eKample, you 
might list all report and product deadlines and Indicate what work Is 
not on schedule or keep a log of how much time Is spent on budgeting 
or editing - maybe the crew needs to hire a budget manager or 
technical writer* 

All So you just sort of provide data to program managers to help keep 

things going In the right directten and as efficiently as possible, huh? 

Evei Eight. Aside from that kind of evaluation, I spent much of my time 
evaluating and providing information to developers, program manage- 
ment, and interested p/^rsona In the field on product specifications. 

All You mean you evaluated the description of the product that appeared 
in the funding contract? That doesn't seem to make any sense to 
me - why waste your time doing that? 
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Evei No, I reviewed and had others review the speclfioatlons for the 

product as they were created and revised by the developers. This 
gave the developers continual feedback for revising and shaping the 
product, Eva Baker, In the Second Handbook of Research on 
Teaching ^ has a good chapter on product development* She talks 
about the initial stages of it as Involving the specification of Instruc- 
tional outcomes, the identification of skills required to get to those 
outcomes and then the designing of strategies for teaching the sldlls. 
The product specifications Include descriptions of the responses 
desiredp content specifications, performance standards and group 
performance levels wanted, and so on. These specifications help the 
developers delimit the attributes of the intended product and promote 
efficient Instruction while sometimes providing a basis for evaluating 
the product later. 

Ah Well how did you actually evaluate the specifications? 

Evei They ware reviewed with respect to how feasible and realistic they 
were in terms of past developmental efforts elsewhere suid relev^t 
evidence In the research literature. 

Ah But I di^'t think there was muchj if any, research that would tell 
you how best to develop a product. 

Eves That*s true. Many writers discuss what they think development 

should look like and, although there^s not much research, it^s help- 
ful to be aware of the approaches they suggest. Sometimes a funding 
agenoy will prefer a particular style of development ^d even use it 
as a criterion in evaluating the product. 

Ah Well, how did you review the product descriptions and development 
style? " 

Evei I was concerned with whether I, or anyone, would be able to evaluate 
the produet later on and give the developers good measurement litfor- 
matlon OR how close they were coming to meeting their objectives. 
Of course, I didn^t actually do all of the reviewing myself, the 
developers and T chose some content experts, potential users, and 
people who had developed similar products in the past, to help us* 
I was just responsible for initiating it and seeing that it got done* 

Ah I guess I don't know much about how you*d evaluate needs and goals 
and product speolflcations. Besides, the developers I work with 
don^t use objectives and speeifications very much. They have a 
much more Intuitive, free-wheeling style of development. ' 
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Evei Yes, but they probably have some mechanism for narrowing their 

work doym to a specific product outline, and you could provide feed- 
back on how closely it matches past promises and future hopes* 
They may need to change some of their Initial plans, and you could 
provide information to help them make the right decisions* Ansrway, 
Eva Baker's chapter is one place to start looking for ideas, and 
there are two extensive works by Sanders and Cunningham ® that 
re^dew a whole set of teohniques for conducting evaluation work like 
this during the formative stages of product development* They talk 
about CKpert reviews, logical analyses, literature reviews, compara- 
tive analyses and surveys, ^and questionnaire data. You might also 
take a look at Bob Stake's comprehensive article on objectives and 
judgment data. He discusses different kinds of judgment data, how 
to collect and use and report them* I think you'll find those papers 
useful later on In your evaluation, too when you've got a product. 

All I have seen Stake's paper and was tiding to use It to help me review 
some of the developers' objectives, but they seem to have a lot 
more objectives than I can find wrttten down anywhere and what Is 
written Just doesn't seem to get at the heart of what they're trying 
to do. 

Evei I know, I had the same problem when I first started. Actually, I 
found several different ways to get at implicit objectives like ha\ang 
reviewers Infer them from theory papers and so on. Of course 
you've got to decide how Important stated objectives are to the actual 
development process so you don't spend too much or too little time 
on them. 

Ah Well, I probably won't be doing much more than I have already* 
I'm glad to have It done* 



Ever It may really never actually be done, Mike Scrlven says that you 
probably should re-evaluate the product goals several times over 
the course of the development of the product to assess the project's 
chances of achieving them given the stage of their current aetlvitles 
each time. He suggests re-analyElng the goals to point out disc rep- 
Mcles between the goals of different groups, between the project's 
goals and its achievements, between the goals of the project and 
the funding agency, and between the goals as spelled out In the 
initial proposal and as later operatlonallged In the project* 

All That sounds like an awful lot of work. 

Eve: One easy way I have found to do It is to make good use of Scrlven's 
; emphasis on critical competitors - you know, the products a user 
might buy Instead of yours. Even before we had a product, I started 
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looking for critlgal competitors. By keeping the developers aware of 
what potential competitors might be, I was able to help them reeog" 
nize what others were and were not capable of acoomplishing. That 
way they could be clear if particular objectives were high-risk ones - 
like there being some strong chance they might fail to reach them. 
Attending to potential competitors also helped me to review and 
evaluate their product specifications more easily, I could then evalu- 
ate their work with respect to what others were finding was feasible 
to produce and saleable on the market, what implementation problems 
similar packages were havings and so on. This also kept me better 
aware of when new products come on the market and changed our 
chances for success or when similar products were being buffeted 
by political winds suggeating possible modifications in our own prod- 
uct* It took some tlmcp though. 

Ah I guess I can see why you were so busy during the first few months, 
I haven't been doing much of this kind of work or even thi^ing about 
it. Instead, I hava started writing evaluation desipis for testing the 
product, I have reviewed some of the evaluation models in the 
Worthen and Sanders book^ and have decided we need a good deoiBion 
oriented design with as much experimentel control as we can get, 

Evai That book does have a pretty complete assortment of evaluation 
models, but I'd be reluctant to just review It and choose one, 
especially at this early stage of the development of your product* 

Ah Why? What's wrong with that? 

Evei Weill the shape of your product, and maybe even the development 
effort Itself probably Is still pretty undefined at this point. Bob 
Stake*^ has cautioned against getting an early preconceived notion of 
success and letting a few objectives or some apparently relevant 
instrumentation distract you from the things that most cone 
people involved in the pr^ess* He suggssts what he calls responsive 
evaluation S which orients you to program activities more than Intents, 
with considerable attention paid to responding to audience requirements 
BMd representing differing value perspectives relevMt to an evaluation 
of the product. Or perhaps maybe you should t^ some goal free 
evaluation like So riven® su^ests where you take a look at possible 
product outcomes before Imowing much about product goals and 
objectives* You might t^ that for a while before you get too 
deeply involved in the project Itsolf, You can't do it later because 
you^ll be too; involved with the developer's intents by that time. This 
way, If you do decide to have someone do a goal free evaluation on 
your project further down the line, you'Jl probably be better able to 
make good use of the work since you willi have been through the 
same thing yourself. Besides, you might get some unpredictable 
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and useful insights now that could really help the developers at a 
time when they have the most latitude in their work. 



Ah That sounds 0K| but I think our funding agent is going to want hard 
data ffid probably objeatlves-based evaluation with pretty conventional 
e^erimental desl^s* 

Eve^ That may well be^ but right now you probably have the most leeway 
you^re going to have and at this early stage you need to retain con- 
siderable flejdblllty and monitor changes of direction both internal 
and external to the project. Your assessment of what's needed may 
be accurate now, but If you get locked in too early, conditions could 
ch^ge and you'd end up with an inappropriate, or at least unsatis- 
fying^ design for some people. You should probably consider a goal 
free or responsive or transactional approach for an early stage of 
evaluatton since they allow you to retain cohslderable fleribility in 
your work, If^ in addition, you keep attuned to the changing political 
winds, project intents, and contractual obligations, then you can allow ' 
your evaluation work to mature with the chanpng program . mid evalua- 
tion contexts Obviously at some point you may need to begin serious 
narrowing and defining of your evaluation work and maybe you shouldn't 
wait too lon^ to begin that since it does take som*e time* Perhaps 
you will need an experimental deslpi or objectives-based orientation, 
but perhaps not. My point is that it is easier to adopt that from 
some of these other orientations, such as goal free, responsive, or 
transactional than it Is to pull back from an objectives-based design 
to a more open, less preordinant procedure. 

All That sounds like good advice, but I don't know if it would work for 
me. But what's transactional evaluation - I don't think I've ever 
heard of it. 

Eve. Well, one of the big pluses of ^al free evaluation is that it allows 
you to get a look at a product without being , ised by knowledge of 
what intents developers had in mind in creatln^^ it* On the other 
hand, responsive evaluation can be employed when you are veiy 
concerned about honoring different value perspectives and being able 
to Gommunicate evaluation activitlei and results to various divergent 
audiences. Communication is probably one of its strong sidts^: 
Transaotlonal evaluation is directed not so much to assessing the 
instructional impact of the product as to evaluating the organizational 
change resulting from innovation. It focuses more on orgMilzatlonal 
dysfunctions that can occur as a result of innovation-induced threats 
to role stability* Robert Rippey is probably the best known pro- 
ponent of transactional evaluation, You might use some of its 
techniques when you are not only concerned about prodding informa- 
tion on product improvementi but also need information about imple- 
mentlng your product in the schools aiid the reslstMce or problems 
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you are likely to have. It helps you to study the dynamics of imple- 
menting your product in the schools. Doesn't the product you work 
on constitute a substantial intervention in the schools? 

Ah YeSj it's going to require the continual involvement of teachers and 
admlnietrators^ as well as some community groups. 

Eve: You might oonslder trying transactional evaluation techniques then, at 
least for a while. You might be able to maintain some of them at 
low cost even later on if you do decide to go to^ objectives-based 
product centered approach. But I suppose you are already spending 
a lot of your time in the field* 

Al: Noj not really. We won't be doing any testing for quite a while yet. 

Eve I Well, I hope you don't wait too long like I did. 

Ah Why, what do you mean? 

Evei Sometimes It takes quite a while to set up testing sites ^d establish 
rapport with the schools. Gaining entry into a school system snd 
establishing a good testmg environment can be an involved proooss, 
and when I first started I taaew ve^ little about it. 

Ah I really don't know anything about It either; could you give me some 
help? 

Eve: Sura, we'll need to spend some time later on talking about It, but 
for now you might start by reading some of the work by C las en. 
Miller, and Coniy'^- who studied attitudes of researchers and 
admlniBtrators ooncermng gaining entry into school systems. They 
list elk differaht aspe^ p^lnlng access to schools which you might 
consider. You might also look at a paper by Manohanl^ who sur- 
veyed offlelal policies for data collection and research in large city 
school ^sterns all over the U, S. He summarized the usual require- 
ments for ent^ and the problems that frequently concern adminis- 
trators* 

Ah Good, that should get me started. 

Evei Oh, there is another good paper by Hayman^^ that Just discusses 
some of the basics of establishing good rapport in schools. 

Ah I probably have some e^ra time to work on this since we won't be 
needing to field test for some time. 

Evei You might consider going out to the schools long before that though. 
Have you read the CSE monograph by Wells Hively?^^ 
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Ah No, why? 



Eves He telki about getting feedback early from people Uke teachers aiid 

adtolnistrators with rei^ tb ttheir ideas on the feaslbillly of putting 
products in the Glassrooms,^^^^^ teaching problems, 

: and so on. You cm do tMs; informal^ during the initial eteges of 

- . product developmant and pve some good feedback to developers on 

product speQlficatloneV -Th reviews we / 

have alrea^ talked abouti while est^ 

school people at the same time* Hively points out that yhle can help 
you Istet y/non the school staff gmi give you- good anecdotal ■ Informa-- . 
tion on side effects and product Impact which you can use as a 
partial basis for developing senBitlve lnstru^ They can 

also give you feedback on bite and piecee of the product as it's 
being created - Uke Lewy^^ suggestBi^ So you might get out tato 
the echoole as soon as Is practical - It also helps you define the 
conte^rt of the evaluation. 



All You've used that term valuation context" before. What do you 
mean ^ it? . 

Eve: Weill there are at least two different sets of activities that begin to 
set what I call the "context of the evaluation" and I think they should 
be conducted before, or at least at the same time as, doing pre- 
product evaluation studies, setting up field relations^ and beginning 
to choose evaluation paradl^s to follow. The first set off activities 
involves beginning to clarify some of the expectations for the evalu- 
ation work held by both internal and e3d;ernal groups to the project. 
For example^ on many projects, evaluators don't spend all their 
time evaluating. They are ejected to perhaps do some adminis^ 
trative work or do data analysis for development or dissemination 
purposes even though it may haye no actu^ connection to their 
evaluation work. This is imderstandable - developers don't spend 
all their time developing either. But an evaluator may appear to ; 
^ have a lot of free time on his hands in early development stages 
since he has no product to evaluate, ^d his time may be drained 
away to do tasks extraneous to Ms work. Without some under- 
stimding within the project, then, as to how roles are being defined 
and played, ^ evaluator may And himself perhaps being held 
accountable for aspects of the development work over which he has 
no control or to which he was only proirtdtag colleglal input. Par- 
haps some of the developers are doing a great deal of field rela- 
tions work and would resent an eTO.luator/S intrusion m this area. : 
Some plarifiGation_of why the evaluate be_in the field may 

be needed. : This clarification of expectations at an early stage 
can greatly faGilltate a smooth development and evaluation effort. 
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Eve: Weill there are at least two different sets of aetivitleB that begin to 
set what I call the "context of the evaluation" and I think they should 
be conducted before, or at least at the same time as, doing pre- 
product evaluation studies, setting up field relations, and beginning 
to choose evaluation paradlpns to follow. The first set off activities 
involves beginning to clarify some of the eKpectatione for the evalu- 
ation work held by both internal and eternal groups to the project. 
For example j on many projects, evaluators don't spend all their 
time evaluating. They are ejected to perhaps do some adminis^ 
trative work or do data analysis for development or dissemination 
purposes even though it may haye no actual connection to their 
evaluation work. This is imderstandable - developers don't spend 
all their time developing either. But an evaluator may appear to v 
^ have a lot of free time on his hands in early development stages 
since he has no product to evaluate, ^d his time may be drained 
away to do tasks extraneous to his work. Without some under- 
stimding within the project, then, as to how roles are being defined 
and played, ^ evaluator may And himself perhaps being held 
accountable for aspects of the development work over which he has 
no control or to which he was only prodding coUeglal input. Per- 
haps some of the developers are doing a great deal of field rela- 
tions work and would resent an cTO-luator's intrusion m this area. 
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be needed. . This clarification of eKpectations at an early stage 
can greatly faollltate a smooth development and evaluation effort* 
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Outside toterests,_ such as profes 

the toding agency may a have Qertam e^eotrtlons : about the 
nature of the evaluation to be conductad. Sometimes these electa- 
tlons are not Glearly stated and have to be Inferred,** Other times 
there are specific erlterla available that Indl^^ how the evaluation 
work i s going to be assessed. 

Ai* You mean like the criteria for meta-evaluation in the PDK book on 
evaluation 7-'-^ 

Evei Yes, if they are particular^ relevant in your case, or at least there 
are people who think they are* It's those kinds of things which help 
you determine what criteria are going to be used to evaluate the 
nature and quali^ of your evaluation work* They help you shape 
your work to be credible, and, therefore, perhaps more useful, 

Al; What about Scriven's Product Checklist?"^ 

Ever Yes, that's the kind of thing I mean. It has been used by federal 
funding agencies as a general guide for evaluating the adequacy of 
product evaluation work - you can see its influence in recent check- 
lists. These documents not only help you judge what kinds of evalu- 
: ation work will have some impact outside the pro jectp but can serve 

as a planning aid, too* Some people have suggested that you might 
just keep folders on each of the items Included in the checklists 
such as need, side effeota, costs, educational si^iflcance, effective- 
ness, ease of implenientation, and so on, and encourage people like 
developers, teachers, and field personnel to send o'^Qu an3rthlng t 
seems appropriate. Thus when you need a quick statement of all 
information to date on a particular item you could pull from your 
own work md the file to produce a fast report. It could keep you 
flexible and current. 

All It also might help you just collect a lot of Irrelevaiit, poor quality 
information. 

Eve! That's true, it needs to be done carefully and with some thought. 

Ah Other than clarlft^g expectations, how else do you determme your 
^ ^'evaluation conte^rt?" ^ 

Ever You can also define the context of your evaluation by addressing 

some of the many evaluation decisions you have to make. Decisions 
- Jike: How do I determine the characteristics of the target groups of 
= interest, including their entry and erit characteristics? How c^ I 

begin to document and describe the environment where the instruc- 
tional training Is to take place? What potential side effect variableB 

- : -■- T4-.-V- - - - — _ 
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and mhiblting or faoilitating constraints on the instructionB should I 
begin to look at? How do i;be^ to contact relevatrt^^^^ ' 
learn of their IMormation needs, timelines for iMormation, and mos^^ 
approprlata reporting mechanisms for them? ^^^ H^ do I determine 
what criteria are likely to be relevant in jud^ng the worth of the 
product, who holds such criteria and how do I operationallM and 
measure along their value dimensions? How do I be^ identifying 
which information will be acceptable to which audiences? As these 
questions get more specific, of course, you frequently are back to 
the stage of working with developers in looking at the product 
speolflcatlons and intents. Sometimes you go by lo^cal analysis^ 
other times you use empirical trial and error. But, ae you work 
through these questions and decisions, you bepn to define the nature, 
constraints, and focus of the evaluation, that is, you put it in an 
evmluatlon context. This enables you to go on to develop management 
and budget plans, reporting and data-^gathering timelines,, and to con- 
duct more formal evaluation studies, if appropriate, using specific 
models, and so on* Determining the conte^rt is probeJbly the hardest 
part of the job and it usually needs to be done before the product 
gets too far developed, when you tove to start tackling sampling. 
Instrumentation, and data ^alysis problems. Also, if you have done 
a good job initially. It ''s easier to detect when the context changes, as 
could easily happen if you are involved in a long project. 

Boy, I can sure see now how you could keep busy until the product 
gets rea% to test! In fact, I'm sure now that I don't have enough : 
time left to do everything that needs to be done before begtontog to 
test the product* Thanks, Eve V I thinks I feel swamped now with 
work. I almost wish I hadn't stopped by. It was a lot easier just 
thinking that all I had to do was wait for the product, f 

Probably the first thing you should do now is develop some priorities. 
I don't taow of any evaluator who completed all this work before a 
product came along - most of us are just happy to get a good part 
of it under control. 



I'd better take a good look at how much time and money I can spend 
on some of these tesks and start to work on the most Important 
ones right away. 

Don't forget to talk to your developers, project mMagers, funding 
agent and even users about some of these tasks. Their input can be 
useful in helping you set your priorities too. 

OK, thmks for all the help, Eve. _ / _ _ - 

Any time, stop by again when you get a free minute. 



Postscript ' - 

Thus there appear to be many potentially profitable ways for an 
evaluator to spend Ms time even before he begins to directly assess the 
quality of an instruetlonal product. Although probably no evaluator ever . 
has the time and resources to do all the thtogs mentlonad abovsi they iriay 
be worth considering as one begins to evaluate products which are in the 
initial stages of development. 
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